

Can summer math programs improve math achievement 
and readiness for algebra content? This randomized 
controlled trial, conducted in collaboration with the 
Silicon Valley Research Alliance, shows that a 19-day 
summer math intervention for students entering grade 8 
improved their scores on the Mathematics Diagnostic 
Testing Project’s Algebra Readiness test by 4 points 
(0.7 standard deviation) and increased estimated 
algebra readiness rates from 12 percent to 29 percent. 
The program did not significantly improve students’ 
interest in math or their sense of competence in math. 
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Summary 


Success in middle school math is a key predictor of students’ success in high school and 
beyond — and is therefore an important issue for policy and practice in California and 
throughout the country. Middle school math coursetaking and success have clear corn 
sequences for the extent to which students reach advanced math courses — such as pre- 
calculus, calculus, trigonometry, or Advanced Placement math — before graduating from 
high school. Completing these advanced math courses can predict how well students are 
prepared for postsecondarydevel math and whether they will be able to participate and 
succeed in regular college math courses without remediation or participation in develop- 
mental math courses. 

Mastering algebra is a critical step to enabling students to succeed in a college preparatory 
math sequence. But many students are unprepared to succeed in algebra, and they fail the 
course the first time they take it (Balfanz, McPartland, & Shaw, 2002; Finkelstein, Fong, 
Tiffany-Morales, Shields, & Huang, 2012; Huang, Snipes, & Finkelstein, 2014). The con- 
sequences of failing algebra can be considerable: only one in five students who fail algebra 
in grade 8 and repeat it in grade 9 achieves proficiency by the end of grade 9 (Finkelstein 
et al, 2012). Students not achieving algebra proficiency by the end of grade 9 have little 
chance of reaching and succeeding in advanced college preparatory math courses by the 
end of high school (Schiller & Muller, 2003; Spielhagen, 2006; Schiller, Schmidt, Muller, 
& Houang, 2010). 

To raise math success rates in middle school, many schools and districts have implemented 
summer math programs designed to improve student preparation for algebra content in 
grade 8. However, little is known about the effectiveness of these programs. While students 
who participate typically experience learning gains, there is little rigorous evidence eval- 
uating the effects of the programs on math achievement or readiness for algebra content. 
This study fills that void by rigorously examining the effects of one such summer program 
on student achievement. 

Elevate Math is a math support program designed by the Silicon Valley Education Foun- 
dation as part of its ongoing effort to help students succeed in middle school math and to 
master important math and science skills that are needed to succeed in college and the 
labor market. Though the program is a year-round effort, its core is an intensive 75-hour 
(19 days over four weeks) summer preparatory course. In summer 2014 the foundation, 
Regional Educational Laboratory West, and several Silicon Valley school districts collab- 
orated on a randomized controlled trial to assess the effects of the Elevate Math summer 
program on math achievement, algebra readiness, and attitudes toward math. Students 
were randomly assigned to a treatment group that received access to the program at the 
beginning of the summer or to a control group that received access to the program later in 
the summer. End-of-program test scores and survey responses of students in the treatment 
group were compared with those of students in the control group prior to their exposure to 
the program. 

The two main findings were: 

• The Elevate Math summer program significantly improved math achievement and 
algebra readiness. Compared with students in the control group, students in the 
treatment group scored significantly higher (4 points, or 0.7 standard deviation) on 


a test of algebra readiness. Students in the treatment group were also significantly 
more likely (29 percent versus 12 percent) to reach achievement thresholds associ- 
ated with success in algebra I. 

• Despite significant positive effects from the program, most students were still 
not ready for algebra I content. On average, students in the treatment group still 
answered most items incorrectly on a test of algebra readiness (scoring 21 out of a 
possible 45). And only 29 percent of students in the treatment group reached an 
achievement level that would predict a better than 50 percent chance of succeed- 
ing in an algebra I course. 

These findings suggest that the Elevate Math summer program improves student math 
achievement and mitigates some summer learning loss. The program’s substantial effect 
on students’ math achievement appears to be driven both by achievement gains over the 
course of the program and by avoiding a decrease in achievement due to inactivity and a 
lack of participation in math instruction over the summer. This suggests that programs 
such as the Elevate Math summer program can be effective in preparing students for 
grade 8 math content. But the fact that on average participants still answered nearly half 
the test items incorrectly suggests that most targeted students will need more support than 
the Elevate Math summer program provides in order to ensure success in algebra. 

Nevertheless, math achievement can be improved and algebra readiness rates can be more 
than doubled over the course of a relatively short period of time. Based on this result, 
an important question is whether additional support during the school year or greater 
amounts of instruction during the summer might further help students reach the levels of 
preparation required to succeed in grade 8 math. 
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Why this study? 


Over the last decade there has been a dramatic increase in the proportion of students 
enrolled in algebra I. This enrollment rise has been due mostly to an interest in having 
students complete key courses like algebra I early enough that they can take advanced 
courses such as precalculus, calculus, trigonometry, or Advanced Placement math before 
graduating from high school. 

Importance of learning algebra in grade 8 

Math courses in U.S. high schools are generally organized in a hierarchical sequence 
(Riegle-Crumb, 2006; Riegle-Crumb & Grodsky, 2010). Research has documented that dif- 
ferences in ability to meet course prerequisites are among the major sources of disparities in 
students’ math progress (Schiller et al., 2010). The sequential nature of math means that the 
timing of key prerequisites (for example, some courses must be taken early in middle school) 
has clear consequences for how far students can progress in the subject during high school 
(Schiller & Muller, 2003; Spielhagen, 2006). Many students who finish middle school are 
not prepared to succeed in a rigorous sequence of college-preparatory math courses in high 
school (Balfanz et al., 2002). Students who do not take algebra I or geometry early enough 
will have little chance of reaching advanced math courses by grade 12. 

For over a decade accountability policies in California increased the pressure for schools 
and districts to ensure that as many students as possible pass algebra I by the end of grade 8 
(Williams, Haertel, & Kirst, 2011; Finkelstein et al., 2012). Until recently, schools were 
penalized on their Academic Performance Index calculation by one performance level 
each time a grade 8 student took the General Mathematics California Standards Test 
(CST) instead of the Algebra I CST. For instance, if a grade 8 student scored proficient 
on the General Mathematics CST, a school would get credit for a score of basic in the cal- 
culation of its Academic Performance Index (California Department of Education, 2011, 
pp. 38-39). 

The challenge of succeeding in algebra I in grade 8 

While the dramatic increase in algebra I enrollment in grade 8 has resulted in greater 
percentages of grade 8 students scoring proficient or advanced on the Algebra I CST, it 
has also led to larger numbers of grade 8 students scoring far below basic or below basic 
(Williams et al., 2011). 

The consequences of failing algebra I can be considerable. Success in advanced math courses 
in high school predicts postsecondary success and careers in science, technology, engineer- 
ing, and math (Adelman, 1999). There is also a close connection between success in middle 
school academic experiences and subsequent performance in high school (see, for example, 
Oakes, Gamoran, & Page, 1992; Stevenson, Schiller, & Schneider, 1994; Wang & Gold- 
schmidt, 2003). In particular, only one in five students who fail algebra I in grade 8 and repeat 
it in grade 9 achieves proficiency by the end of grade 9 (Finkelstein et al, 2012). And only 
16 percent of students that receive a C or below in algebra I in grade 8 enroll in geometry 
in grade 9 (Finkelstein et al, 2012). In short, few students recover from failing algebra I, and 
failing the subject in grade 8 or 9 disrupts their progress, substantially reducing the likelihood 
that they will enroll and succeed in the higher level courses required for college success. 
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A central objective of the Common Core State Standards, in which California partici- 
pates, is to greatly increase math proficiency among students graduating high school in 
order to greatly decrease the need for remediation in postsecondary education. Changes 
resulting from the standards mean that for many California districts algebra I courses have 
been replaced with integrated math courses that include algebra and other content. Key 
questions will be how to ensure that students are prepared for algebra content and how to 
maximize the proportion of students who pass middle school math courses that include 
algebra content the first time students take them. 

The desire to increase the number of students succeeding in algebra has led schools and 
districts to implement summer programs for students at risk of failing. In the past three 
years the San Diego Unified School District has enrolled at-risk students who will start 
algebra I in the fall into Pre-Algebra Upgrade, a learning module, for summer school 
(Learning Upgrade, 2012). In 2013 Virginia Beach City Public Schools offered two summer 
programs to strengthen students’ pre-algebra skills and help them gain confidence for 
algebra I: Algebra Readiness Academy for rising grade 9 students and Alge-Prep program 
for rising grade 7 and 8 students (Virginia Beach City Public Schools, 2014). In Silicon 
Valley several districts participated in Stepping Up to Algebra (a precursor to the Elevate 
Math summer program that is the subject of this study), which was designed to improve 
student preparation and success in algebra I. 

Little is known about the effects of these programs. Some evidence from nonexperimental 
comparison group studies indicates that students who participate experience gains in math 
achievement, but there is little rigorous experimental evidence assessing the programs’ 
effects on math achievement or success in subsequent algebra I courses. 

The Elevate Math summer program 

Elevate Math is an intervention program that helps incoming grade 8 students succeed 
in algebra and related content that aligns with the Common Core State Standards in 
math (see box 1 for more details about the program). Designed by the Silicon Valley Edu- 
cation Foundation as part of its ongoing effort to help students master important math 
and science skills needed to succeed in college and career, the program is a year-round 
effort that includes support for math performance. The core of the program is an intensive 
75-hour (19 days over four weeks) summer preparatory course taught by a certified teacher, 
along with support from a college-educated teaching assistant (see figure 1 for the logic 
model underlying the Elevate Math summer program). In 2012/13 more than 800 students 
in 18 districts in Santa Clara County, California, participated in the Elevate Math summer 
program. 

The Silicon Valley Education Foundation works with participating school districts to iden- 
tify grade 7 students from each district who, based on their prior CST scores, may need 
additional support to succeed in grade 8 math. The foundation then reaches out to these 
students and their parents (through parent information events, emails, and teachers and 
counselors) to explain the program and its benefits and encourage students to participate. 

Elevate Math targets grade 7 students who scored at the high basic level or the low profi- 
cient level on the grade 6 math CST. 1 In the past, students with these scores were placed 
into algebra I in grade 8, 2 but they were unlikely to succeed (that is, to score at or above 
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Box 1. About the Elevate Math summer program 


The Elevate Math summer program consists of four main components: 

• A Common Core State Standards-based curriculum that covers four math content 
modules: properties and operations, linear equations, ratios and multiple representations, 
and transformational geometry. 

• Approximately 19 days of four hours of blended learning classroom instruction, with one 
hour each day spent on Khan Academy (a free online learning system with thousands of 
educational resources). Each Khan Academy session includes a set of computer-based 
exercises that reflect the topics covered in the classroom that day. Students also have 
access to Khan Academy web-based videos to review any math topics covered during their 
class time. 

• A field trip to a local college or university, as well as a college information night for families 
and students to encourage college awareness. During the study period, most students 
(74 percent) attended the college field trip. 

• Credentialed teachers and their college-level teacher assistants receive 40 hours of 
Common Core State Standards-based professional development provided by the Santa 
Clara County Office of Education and the Krause Center for Innovation. The first 24 hours 
(which occur prior to the summer instruction) include training on curriculum understanding 
and implementation, instructional strategies aligned with the standards, math practices, 
technology integration in the classroom, and student engagement. The next 16 hours 
(which occur over the summer after instruction has begun) are spent in a professional 
learning community setting, where a coach facilitates the meeting to provide a better 
understanding of specific Common Core State Standards instructional strategies and 
math practices that are useful to teach the Elevate Math curriculum. 

The Silicon Valley Education Foundation partners with other nonprofits and works closely 
with 18 school districts and the Santa Clara County Office of Education to offer Elevate Math, 
which consists of the Elevate Math summer program as well as year-round math supports. Tech- 
nology-based firms in the area also provide resources to the foundation to support these student 
programs. According to estimates from the foundation, the Elevate Math summer program costs 
$500 per student (based on an average class size of 30); this estimate includes the costs 
of a credentialed teacher, a college-level teaching assistant, a college field trip, and Common 
Core State Standards-based curriculum and professional development for teachers and the col- 
lege-level teacher assistants. Schools provide the classroom and a site principal. In addition to 
these costs, laptop computers are provided to every student through a donation from Dell Wyse. 
See http://svefoundation.org/programs/elevate-math for more information on Elevate Math. 


proficient on the Algebra I CST in grade 8) without additional support: in 2013 only 
50 percent of students who took algebra I in grade 8 scored at the proficient level (Cal- 
ifornia Department of Education, 2013). Silicon Valley students who scored at the low 
proficient level on the grade 6 math CST had a 50 percent chance or less of succeeding in 
algebra I in grade 8 (Huang et al, 2014). Students had a 50 percent chance or less of sue- 
ceeding in algebra I in grade 8 unless they scored at least 17 points (.27 standard deviation) 
above the threshold for proficiency on the grade 6 math CST (Huang et al., 2014). With 
the changes in courses associated with the Common Core State Standards, students who 
score at the high basic or low proficient level are now likely to be placed into Common 
Core math 8 courses, which integrate algebra and other topics. Some have argued that 
math 8 courses contain more advanced content than existing algebra I courses in grade 8. 
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Figure 1. Elevate Math summer program logic model 


Program inputs 


Intervention 


Mediators 


Student-level outcomes 


Instructional staff 

• Credentialed teacher 

• Half-time teaching 
assistant 


Elevate Math professional 
development (24 hours) 

• Curriculum 

• Instructional strategies 

• Student academic 
mindsets 

• Classroom management 

• Technology 


Elevate Math professional 
development with coach 
(16 hours) 

• Biweekly meetings to 
discuss instructional 
strategies 

• Classroom observation 
with feedback 


Elevate Math summer 
session (four hours per 
day for 19 days) 

• Four instructional 
modules: 

• Properties and 
operations 

• Linear equations 

• Ratios and multiple 
representations 

• Transformational 
geometry 

• One hour per day of 
Khan Academy 
computer-based 
exercises 

• College awareness 
component that 
includes a field trip to a 
local college or 
university and a college 
information night for 
students and families 


Additional time on task 
for math instruction 

Personalized support 
from credentialed 
teacher and teaching 
assistant 

Access to 
computer-based 
instructional support 


Increase students’ 
interest in math and 
self-confidence in their 
ability to do math 


Improved student math 
skill and algebra 
readiness (measured by 
the Mathematics 
Diagnostic Testing Project 
Algebra Readiness test) 


Increased success in 
grade 8 math courses 

Increased enrollment in 
higher level math classes 


Not measured in this study 


Source: Authors' compilation. 


To the extent that the math 8 courses contain content similar to and at least as challeng- 
ing as that in algebra I courses, students scoring at the high basic or low proficient level 
may be unlikely to succeed in the new math 8 courses without additional support. 

In addition to prior preparation, student success in grade 8 math may be affected by summer 
learning loss. Summer learning loss in math has been well documented. Previous studies 
have shown that students experience summer learning loss, that it is more pronounced 
for math than for reading, and that it is more pronounced among lower income students 
(Cooper, Nye, Charlton, Lindsay, & Greathouse, 1996). Summer enrichment programs can 
ameliorate summer learning loss (Chaplin & Capizzano, 2006). In general, previous research 
indicates that the majority of rising grade 8 students do not participate in summer school 
(U.S. Department of Education, National Center for Education Statistics, 1999; Terzian, 
Anderson, & Hamilton, 2009). To fill all available slots in the Elevate Math summer 
program, the Silicon Valley Education Foundation has had to continue recruitment efforts 
through the first day of the summer program. Thus, the study team hypothesized that the 
Elevate Math summer program serves students who would otherwise be unlikely to continue 
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formal study in math over the summer. This is substantiated by survey data indicating that 
87 percent of students randomly assigned to a control group that did not have access to the 
Elevate Math summer program for the first half of the summer did not participate in other 
summer learning while waiting for their four-week Elevate Math summer program session to 
begin. As these students are typically not participating in math academic enrichment, they 
are susceptible to summer learning loss during the first four weeks of the summer. 

The summer program provides these students a bridge into the beginning of the fall 
semester of grade 8. A key question then is whether the Elevate Math summer program 
and others like it can improve student math achievement and algebra readiness over the 
summer before grade 8. This report presents the results of a randomized controlled trial 
that addresses this question. 


What the study examined 


Though previous research on the Elevate Math summer program has found positive pre- 
and post-program gains in math proficiency, as well as positive attitudinal changes toward 
math, the Elevate Math summer program has not been subjected to a rigorous test of its 
effects on math achievement, algebra readiness, or student attitudes toward and engage- 
ment in math learning. Without comparing a set of Elevate Math summer participants 
and a set of equivalent students who did not take part in the program, it is difficult to 
ascertain whether the gains are indeed caused by the Elevate Math summer program. Thus 
the Silicon Valley Research Alliance and Regional Educational Laboratory (REL) West 
collaborated to mount this rigorous randomized controlled trial to answer three research 
questions: 

1. What is the impact of the Elevate Math summer program on the math achievement 
and algebra readiness of rising grade 8 students? 

2. What is the impact of the Elevate Math summer program on math achievement in the 
math topic areas most closely aligned with the program’s curriculum? 

3. What is the impact of the Elevate Math summer program on the math interest and 
math self-efficacy of rising grade 8 students? 

See box 2 for a summary of the data and methodology used in the study and appendix A 
for more details. 

To contextualize the analysis of program effects, REL West also assessed the implementa- 
tion of the Elevate Math summer program. The assessment focused on five program com- 
ponents: students’ attendance at the summer program, student participation in the college 
field trip, instructional time spent on Khan Academy, teachers’ attendance at professional 
development sessions, and teachers’ completion of each module of the program curriculum. 
The goal of the analysis was to assess the extent to which the core program components 
were implemented as designed, so that readers can assess whether the evaluation presents 
a fair test of the program. 

The findings of this study will help the Silicon Valley Education Foundation refine its 
Elevate Math summer program to better serve participating districts and their students as 
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Box 2. Data, outcomes, and methodology 
Data 

The randomized controlled trial was conducted in summer 2014 at eight schools in six dis- 
tricts in California’s Silicon Valley (the Morgan Hill Unified School District and five of the seven 
feeder districts to East Side Union High School in San Jose). Participating districts identified 
eligible students based on existing grade 6 California Standards Test (CST) data. Though the 
Elevate Math summer program targeted students who scored at the high basic level or the low 
proficient level on the grade 6 math CST, all 496 grade 7 applicants from participating districts 
were accepted, regardless of prior math skills. More than half of participating students fell into 
the target range for the intervention based on their grade 6 math CST scores (see table). 


Grade 6 math California Standards Test performance levels of study participants 



Far below 

Below 

Basic 

Proficient 




basic 

(scale 
score of 
150-252) 

basic 

(scale 
score of 
253-299) 

Low 

(scale 
score of 
300-324) 

High 

(scale 
score of 
325-349) 

Low 

(scale 
score of 
350-360) 

High 

(scale 
score of 
361-414) 

Advanced 

(scale 
score of 
415-600) 

Unknown 

Percent 

0.84 

8.60 

19.92 

34.80 

17.82 

14.05 

2.52 

1.47 

Number 









(n = 477) 

4 

41 

95 

166 

85 

67 

12 

7 


Note: Percentages may not sum to 100 because of rounding. 
Source: Authors' analysis of primary data collected for the study. 


Outcomes 

The study examines the Elevate Math summer program’s effects on three sets of outcomes: 
math achievement and algebra readiness, achievement in specific math topic areas that 
appear to be aligned with the Elevate Math summer program’s curriculum, and math interest 
and math self-efficacy. 

Math achievement was measured using the Mathematics Diagnostic Testing Project 
(MDTP) Algebra Readiness test, which was administered to the treatment and control groups 
on the first and last days of their participation in the summer program. The test consists of 45 
multiple-choice items in seven topic areas (see table A6 in appendix A). Previous research has 
shown that this test is highly predictive of success in algebra I in grade 8 (that is, scoring pro- 
ficient or higher on the Algebra I CST in grade 8) and that grade 7 students had to pass three 
or more MDTP topic areas to have a greater than 50 percent chance of success in algebra I in 
grade 8 (Huang et al., 2014). Therefore, for this study, math achievement was defined as the 
MDTP Algebra Readiness total score, and algebra readiness was defined according to whether 
students passed three or more MDTP topic areas. 

Of the seven topic areas assessed by the MDTP Algebra Readiness Test, the three that 
most closely aligned with the Elevate Math summer program’s curriculum were decimals, their 
operations and applications, and percent; literals and equations; and geometric measurement 
and coordinate geometry. 

Math interest and math self-efficacy were assessed using measures drawn primarily from 
a student perception survey developed by Gilbert (2008) and administered by the Silicon Valley 
Education Foundation as part of its previous and ongoing program monitoring efforts. The 
math interest scale consisted of five items related to students’ interest in math (for example, 
how exciting math is to the students, how much the students like doing math). Each item 

(continued) 
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Box 2. Data, outcomes, and methodology (continued) 


asked students to rate their responses on a five-interval scale ranging from 1 (not at all true 
for me) to 5 (very true for me), with the midpoint (3) indicating somewhat true for me. The math 
self-efficacy scale consisted of seven items related to students’ confidence in their ability to 
do math. For example, one item asks, “Even if a new topic in math is hard, how confident are 
you that you can learn it?” The response options are on a scale from 1 (not at all confident) to 
5 (very confident), with the midpoint (3) indicating somewhat confident. 

Methodology 

Participating students were randomly assigned either to a treatment group that was scheduled 
to enroll during Elevate Math’s first four-week summer session or to a control group that did 
not enroll in the first four-week session but was allowed to enroll in the second four-week 
summer session. Random assignment was conducted separately for students in each program 
site. Control group outcomes at the beginning of the second session are valid estimates of the 
outcomes that would have been observed among students in the treatment group, had they 
not had access to the program. 

Baseline equivalence was assessed by comparing performance on the grade 6 math CST 
between the treatment and control groups. Program effects were calculated by comparing treat- 
ment group outcomes at the end of the first summer session to control group outcomes at the 
beginning of the second summer session. Program effects were estimated using a single-level 
regression model comparing average outcomes in the treatment and control groups. The regres- 
sion model controlled for grade 6 CST scores in order to improve the precision of the estimates 
and adjust for any differences in pre-program achievement. Research questions 1 and 3 were the 
primary confirmatory research questions, and the estimates used to address them were adjusted 
for multiple comparisons using the Benjamini-Hochberg (1995) approach. The analysis provided 
in response to research question 2 is considered exploratory, and the associated statistical tests 
are considered less definitive, so tests for statistical significance for these research questions 
were not adjusted for multiple comparisons. The estimated program effects represent a combina- 
tion of two factors: the learning gains experienced by the treatment group during the program and 
the learning loss experienced by the control group during the first part of the summer. 

Appendix A provides more detail regarding the analytic models used to estimate program 
effects, as well as information regarding the extent of student attrition and the characteristics 
of students in the final analytic sample. 


well as districts and students that participate in future years. The findings will also inform 
the efforts of other local education agencies, state education agencies, and private stake- 
holders who are considering strategies for improving middle school students’ math achieve- 
ment and algebra readiness. Moreover, this study serves as an example of how school 
districts and educational support providers can collaborate to conduct rigorous evaluations 
of educational interventions in the course of their normal operations. 

What the study found 


This section presents the study’s findings of the Elevate Math summer program’s effects on 
three sets of outcomes: math achievement and algebra readiness, achievement in specific 
math topic areas, and math interest and math self-efficacy. Information on program imple- 
mentation, which may help contextualize these findings, is in box 3. 
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Box 3. Implementation analysis 


To contextualize the estimated program effects, the study assessed the extent to which the 
Elevate Math summer program components were implemented according to the basic program 
design. The assessment focused on five program components: students’ attendance of the 
summer program, student participation in the college field trip, instructional time spent on 
Khan Academy, teachers’ attendance at professional development sessions, and teachers’ 
completion of each module of the program curriculum. 

The Silicon Valley Education Foundation collected attendance records and records of 
instruction and professional development and administered a weekly teacher checklist and a 
student survey on the last day of the summer program. The weekly teacher checklist included 
items on curriculum and time spent on Khan Academy. The student survey included topics 
such as math interest, math self-efficacy, and different program components (including the 
college field trip). 

Prior to the study launch, the study team, in consultation with the Silicon Valley Education 
Foundation, set thresholds for full implementation. The team defined “full implementation" 
as 80 percent of students attending 80 percent of instruction, 80 percent of students par- 
ticipating in the college field trip, at least 30 minutes of Khan Academy instruction per day, 
80 percent of teachers attending 80 percent of the professional development, and teachers 
covering at least 80 percent of each of the four modules. 

Daily attendance data collected by the Silicon Valley Education Foundation indicate that 
83 percent of students in the treatment group (196 out of 239) attended at least 15 of the 19 
days of instruction and spent an average of 48 minutes on Khan Academy per day. Of the 159 
students in the treatment group who responded to the student survey, 74 percent reported 
attending the college field trip. Eight of the ten teachers (80 percent) attended 80 percent of 
the professional development. And 5 of the 10 teachers (50 percent) responded to the weekly 
teacher checklist and covered 100 percent of the first three modules — which included proper- 
ties and operations, linear equations, and ratios and multiple representations — and over half 
the transformational geometry module. 


The Elevate Math 
summer program 
increased the 
percentage of 
students who 
passed at least 
three MDTP 
Algebra Readiness 
topic areas from 
12 percent in the 
control group to 
29 percent in the 
treatment group 


The Elevate Math summer program significantly improved math achievement and algebra readiness 
among participating grade 7 students 

The Elevate Math summer program improved the math achievement of the treatment 
group compared with the control group across several metrics (table 1). The treatment 
group’s performance on the grade 7 MDTP Algebra Readiness test exceeded that of the 
control group. Out of 45 possible points, the average MDTP score for the treatment group 
was 21 points, compared with 17 points for the control group. This statistically significant 
difference of 4 points is the equivalent of 0.7 standard deviation. 

The Elevate Math summer program also had a positive, statistically significant effect on 
algebra readiness, as measured by whether students mastered at least three of seven MDTP 
topic areas. The program increased the percentage of students who passed at least three 
MDTP Algebra Readiness topic areas from 12 percent in the control group to 29 percent 
in the treatment group (see table 1). This 17 percentage point difference is statistically 
significant and amounts to an improvement of 42 percent, compared to the control group 
(17 minus 12, then divided by 12). 
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Table 1. The Elevate Math summer program had a positive effect on grade 7 students’ Mathematics 
Diagnostic Testing Project Algebra Readiness test scores 



Adjusted mean 





Outcome measure 

Treatment 

group 

(standard 

deviation) 

Control 

group 

(standard 

deviation) 

Difference 

(standard 

error 8 ) 

p-value 

(adjusted 

p-value b ) 

Effect size 8 

Unweighted 
student 
sample size 

Mathematics Diagnostic Testing Project Algebra 
Readiness test total scale score (points) 

20.81 

(6.63) 

16.80 

(5.92) 

4.01*** 

(0.56) 

<.001 

(<.001) 

0.68 

349 

Algebra readiness 11 (percent passing at least 
three of seven topic areas) 

29 

(46) 

12 

(32) 

(4) 

<.001 

(<.001) 

0.53 

349 


*** Significantly different from zero at the .001 level, two-tailed test. 

Note: A regression model with a dichotomous indicator for treatment status was used to estimate average differences between the 
treatment and control group outcomes. The model included prior achievement, as measured by grade 6 math California Standards Test 
scores, and dichotomous indicators representing each school site. The missing-indicator method (White & Thompson, 2005) was used 
to account for missing values on the grade 6 math CST. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. Calculated using the Benjamini-Hochberg (1995) procedure. 

c. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

d. Because the outcome is dichotomous, a logistic regression analysis was also conducted to assess the findings' sensitivity to specifi- 
cation (see table B2 in appendix B). 

Source: Authors' analysis of primary data collected for the study; see appendix A for details. 


The Elevate Math summer program also showed positive effects on the specific topics the 
program targeted: decimals, their operations and applications, and percent; literals and 
equations; and geometric measurement and coordinate geometry (table 2). 


Table 2. The Elevate Math summer program had positive effects on grade 7 student achievement in 
three topic areas of the Mathematics Diagnostic Testing Project Algebra Readiness test 


Adjusted mean 
Treatment Control 

Topic area of the Mathematics group group Difference 95 percent Unweighted 

Diagnostic Testing Project (standard (standard (standard confidence student 


Algebra Readiness test 

deviation) 

deviation) 

error 8 ) 

p-value b 

interval 

Effect size 8 

sample size 

Decimals, their operations and 
applications, and percent score 
(points) 

3.91 

(1.70) 

3.05 

(1.69) 

0.86*** 

(0.17) 

<.001 

0.52 to 
1.20 

0.51 

349 

Literals and equations score 
(points) 

3.65 

(1.51) 

3.13 

(1.49) 

0.52*** 

(0.14) 

<.001 

0.23 to 
0.80 

0.35 

349 

Geometric measurement and 
coordinate geometry score 
(points) 

2.50 

(1.44) 

1.90 

(1.29) 

0.59*** 

(0.14) 

<.001 

0.32 to 
0.87 

0.46 

349 


*** Significantly different from zero at the .001 level, two-tailed test. 

Note: A regression model with a dichotomous indicator for treatment status was used to estimate average differences between the 
treatment and control group outcomes. The model included prior achievement, as measured by grade 6 California Standards Test math 
scores, and dichotomous indicators representing each school site. The missing-indicator method (White & Thompson, 2005) was used 
to account for missing values on the grade 6 math CST. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. Not adjusted to account for multiple hypothesis tests because the estimates were part of the exploratory impacts. 

c. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

Source: Authors' analysis of primary data collected for the study; see appendix A for details. 
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Math scores at the end of the Elevate Math summer program suggested that most students were 
still not ready for algebra I content 


Despite the Elevate Math summer program’s effects, students’ math achievement at the 
end of the program suggested that many students were still not ready for the algebra 
content in grade 8 math courses. The treatment group’s math achievement, as measured 
by the MDTP Algebra Readiness test, exceeded that of the control group. At the end 
of the program, the average student in the treatment group correctly answered 21 out of 
45 items (or 47 percent), and most students in the treatment group (nearly 70 percent) 
had not reached achievement benchmarks associated with having a 50 percent or better 
chance of passing algebra I in grade 8 (that is, achieving mastery in three out of seven 
MDTP Algebra Readiness topic areas; see table 1). 

These findings suggest that, while the Elevate Math summer program had a positive effect 
on student achievement and readiness for algebra content, participating students will most 
likely need more support to succeed in grade 8 math courses with algebra content. 

More than a third of the estimated program effect on Mathematics Diagnostic Testing Project 
Algebra Readiness scores can be attributed to summer learning loss among the control group 

Because students in the treatment group began the Elevate Math summer program earlier 
in the summer than students in the control group, comparing the two groups’ performance 
on a baseline test on their first day of class amounts to a randomized controlled trial of 
the effects of summer learning loss. Because students were randomly assigned to the two 
groups, the baseline scores of the control group represent an experimental estimate of what 
the baseline scores of the treatment group would have been at that same point in the 
summer, had the students in the treatment group delayed beginning the summer program 
for approximately four weeks. 


While the Elevate 
Math summer 
program had a 
positive effect 
on student 
achievement and 
readiness for 
algebra content, 
participating 
students will 
most likely need 
more support to 
succeed in grade 8 
math courses with 
algebra content 


The delay in starting the summer program resulted in a significant reduction in math 
achievement. In particular, the treatment group, assessed in the beginning of the summer, 
had an average MDTP Algebra Readiness test score of 18.28, while the control group, 
assessed approximately four weeks later, had an average score of 16.81, 1.47 points lower 
than the treatment group (these estimates control for differences in grade 6 math CST 
scores between the two groups; table 3). This is approximately 37 percent of the total 
effect of the Elevate Math summer program on MTDP scores. This suggests that, without 
the program the students targeted for the Elevate Math summer program would experi- 
ence deterioration in math performance over the first part of the summer. It also suggests 
that approximately 37 percent of the effect of the Elevate Math summer program can be 
attributed to avoiding summer learning loss among the treatment group. 


The estimated level of interest in math for the treatment group was higher than that of the control 
group, but the difference was not statistically significant; there was no evidence of positive effects 
on math self-efficacy 

In addition to improving math achievement and preparation for algebra, the Elevate Math 
summer program aims to increase students’ interest in math and to improve their sense of 
competence in the subject. The findings on the effectiveness of these efforts are mixed and 
inconclusive. 
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Table 3. Baseline scores on the Mathematics Diagnostic Testing Project Algebra Readiness test were 
about 1.5 points lower for grade 7 students in the control group than for students in the treatment 
group 



Adjusted mean 




Outcome measure 

Treatment 

group 

(standard 

deviation) 

Control 

group 

(standard 

deviation) 

Difference 

(standard 

error 8 ) 

p-value 

Unweighted 

student 

Effect size b sample size 

Mathematics Diagnostic Testing Project Algebra 
Readiness test total scale score (points) 

18.28 

(6.62) 

16.81 

(5.92) 

1.47* 

(0.59) 

0.013 

0.25 342 


* Significantly different from zero at the .05 level, two-tailed test. 

Note: A regression model with a dichotomous indicator for treatment status was used to estimate average differences between the 
treatment and control group outcomes. The model included prior achievement, as measured by grade 6 California Standards Test math 
scores, and dichotomous indicators representing each school site. The missing-indicator method (White & Thompson, 2005) was used 
to account for missing values on the grade 6 math CST. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

Source: Authors' analysis of primary data collected for the study; see appendix A for details. 


Level of math interest, measured on a five-point scale (1, low, to 5, high) using an online 
survey, averaged 2.83 for students in the treatment group and 2.63 for students in the 
control group (table 4). The difference is equivalent to approximately 0.2 standard devia- 
tion and is not statistically significant at the .05 level. 

With respect to students’ sense of math self-efficacy, the estimated difference between stu- 
dents in the treatment and control groups is closer to zero (approximately 0.03 standard 
deviation) and is not statistically significant at the .05 level (see table 4). 


Table 4. The Elevate Math summer program’s effect on grade 7 students’ math interest and math 
self-efficacy was not statistically significant 


Outcome measure 
(1, low, to 5, high) 

Adjusted mean 

Treatment Control 

group group 

(standard (standard 

deviation) deviation) 

Difference 

(standard 

error 8 ) 

p-value 

(adjusted 

p-value b ) 

Effect size 8 

Unweighted 
student 
sample size 

Math interest 

2.83 

2.63 

0.20 

0.10 




(1.12) 

(1.12) 

(0.12) 

(0.20) 

0.18 

329 

Math self-efficacy 

3.21 

3.19 

0.02 

0.80 




(0.90) 

(0.86) 

(0.09) 

(0.80) 

0.03 

329 


Note: A regression model with a dichotomous indicator for treatment status was used to estimate average differences between the 
treatment and control group outcomes. The model included prior achievement, as measured by grade 6 California Standards Test math 
scores, and dichotomous indicators representing each school site. The missing-indicator method (White & Thompson, 2005) was used 
to account for missing values on the grade 6 math CST. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. Calculated using the Benjamini-Hochberg (1995) procedure. 

c. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

Source: Authors' analysis of survey data collected by the Silicon Valley Education Foundation for the study; see appendix A for details. 
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Implications of the study findings 


This study affirms that the Elevate Math summer program improves math skills for stu- 
dents as they approach grade 8. The findings show that the program significantly improves 
math achievement as measured by the MDTP total scale score and that it significantly 
improves the percentage of students that can be classified as “algebra ready.” The findings 
also show that the Elevate Math summer program works both by increasing students’ math 
skills and by helping students avoid summer learning loss through the first half of the 
summer. In fact, 37 percent of the program effect is due to the avoidance of summer learn- 
ing loss among the treatment group. Unfortunately, the design of the study does not allow 
for examining whether the positive effects persist into the fall or beyond. By contrast, the 
findings suggest that the Elevate Math summer program has no effect on students’ sense of 
efficacy in math or their interest in the subject. 

The achievement effects of the Elevate Math summer program are larger than the impact 
estimates from other evaluations. The score differences between the treatment and control 
groups amounted to 4 points, or 0.7 standard deviation. Compared with evidence from a 
large meta-analysis of randomized controlled trials in education, the effect of the Elevate 
Math summer program appears to be twice as large as the typical (median) effect found 
among middle school education interventions (Lipsey et al., 2012, p. 34). 

One concern in interpreting the effect is that it may be easier to improve on diagnostic 
tests than on norm-referenced or criterion-referenced state tests. Therefore, to get a sense 
for how the effect might translate into changes on the types of tests more typically used 
for accountability purposes, the study team examined the correlation between the MDTP 
Algebra Readiness test and the grade 7 math CST in a sample of students from districts 
in the Silicon Valley. 3 For the control group the average MDTP score translates into a 
CST score of 326, 1 point above the threshold for high basic. For the treatment group the 
average MDTP score translates into a CST score of 349, 1 point below the threshold for 
proficiency. This is equivalent to moving students from the 22nd to the 37th percentile in 
the distribution of the grade 7 math CST scores among students in the Silicon Valley. 


While student 
achievement is 
significantly better 
among students 
who participated 
in the Elevate 
Math summer 
program than 
among students 
who did not, it is 
still below typical 
achievement 
for students in 
the surrounding 
districts 


While student achievement is significantly better among students who participated in the 
Elevate Math summer program than among students who did not, it is still below typical 
achievement for students in the surrounding districts. Moreover, the average performance 
of students who participated in the program suggests that most students would not be 
prepared to succeed in algebra I. On average, students in the treatment group failed to 
correctly answer most items on the MDTP assessment. In addition, 70 percent of students 
in the treatment group did not meet the criteria for algebra readiness at the end of the 
program. Huang et al. (2014) suggest that students have to score at least 0.3 standard devi- 
ation above proficiency to have better than 50-50 chance of succeeding in algebra I in 
grade 8. The fact that the treatment group reached a point equivalent to 1 point below 
proficiency suggests that, while the Elevate Math summer program had a meaningful effect 
on student math skills, more improvement would be necessary before students are prepared 
for the higher level math content they will face in grade 8 and beyond. To be fully prepared 
for math courses in grade 8 and beyond, students targeted for the Elevate Math program 
need to combine summer supports with other, longer term, supports for accelerating their 
progress in math. 
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This pattern of findings suggests that summer programs such as Elevate Math may be 
important tools for improving math achievement among students entering grade 8 but 
that they are not sufficient by themselves for ensuring that students are ready for chal- 
lenging middle school math courses. For students in the high basic to low proficient range 
(who were targeted for the intervention) and other students who are motivated to partici- 
pate, the evidence presented in this report indicates that participating in the Elevate Math 
summer program would likely improve their math skills. At the same time, students like 
those in this study, 82 percent of whom scored in the low proficient category or below on 
the grade 6 math CST, would likely not be sufficiently prepared for algebra and would need 
additional support to succeed. This evaluation assessed the effects of the Elevate Math 
summer program only, so further research is recommended to examine the effects of the 
additional year-round math support provided as part of Elevate Math. 

Furthermore, though the implementation of Common Core State Standards is creating 
changes in course sequences and content, to the extent that the courses students take in 
grade 8 still contain substantial algebra content, the pattern of findings described here 
may apply to the new Common Core-aligned courses. The underlying ambition for many 
students who participate in Elevate Math (and their parents) is to reach a level of math 
proficiency that enables them to take a high school course sequence that includes cal- 
culus in their senior year. The findings of this study suggest that some students emerging 
from the Elevate Math summer program will do well in grade 8 courses with substantial 
algebra content. This in turn bodes well for their success in an accelerated math sequence 
that includes calculus. For the students who still struggle with math achievement after 
the Elevate Math summer program, it may make sense to design a course pathway that 
enables ongoing support for these students while maintaining a clear view of the entire 
high school math sequence. 


Summer programs 
such as Elevate 
Math may be 
important tools 
for improving 
math achievement 
among students 
entering grade 8, 
but they are not 
sufficient by 
themselves for 
ensuring that 
students are ready 
for challenging 
middle school 
math courses 


Limitations of the study 


This study has five main limitations. 

First, the study sample is limited to students from the participating Silicon Valley Research 
Alliance districts. Therefore, the findings may be generalizable only to students and dis- 
tricts that are similar to the study sample. 

Second, given the change in curriculum and the lack of existing evidence connecting the 
MDTP Algebra Readiness test to achievement on the (yet to be implemented) Common 
Core math assessments, it is unclear whether a positive effect on MDTP achievement will 
translate into positive effects on achievement in Common Core grade 8 math courses. 

Third, student attrition from the analytic sample does not appear to have occurred at 
random (see appendix A). Students with lower grade 6 CST scores were more likely to be 
absent from the final analytic sample, and this pattern may have been more pronounced in 
the control group than in the treatment group. Though the differences are not significant, 
the baseline grade 6 CST scores in the treatment group were somewhat higher than those 
in the control group. While the analysis controlled for these differences, the possibility 
that unmeasured differences could have affected these estimates remains. In addition to 
the fact that attrition was nonrandom, it occurred for about 27 percent of the sample. To 
the extent that those who left the sample were different from those who remained, the 


13 


generalizability of the findings from this study might be limited. But if students who left 
the sample are not systematically different from those who remained (or if the differences 
do not go beyond the grade 6 CST scores, which are controlled for), the estimates present- 
ed would accurately represent the effect observed among students who participated in the 
intervention. Nevertheless, replicating this study in another sample may be the best way to 
assess the validity of these initial results. 


Fourth, because the treatment group took the MDTP Algebra Readiness test at the begin- 
ning and end of the program, the treatment group’s post-test could have been affected by 
exposure to the assessment four weeks prior to the post-test. Familiarity with the MDTP 
Algebra Readiness assessment might have increased performance and post-program out- 
comes among the treatment group, meaning that the estimated effects of the Elevate Math 
summer program could be overstated. However, previous research on MDTP Algebra 
Readiness test-re test reliability (estimated to be .87) suggests that students who re-take the 
tests after two weeks and up to six months do not improve their scores without intervening 
instruction (California State University/University of California Mathematics Diagnostic 
Testing Project, 1995). As such, improvements due to increased familiarity with the test 
over a four-week span seem to be an unlikely explanation for the improvement in test 
scores. Moreover, a test-retest effect would not explain the 37 percent of the program effect 
that was due to the avoidance of summer learning loss in the program group. 

Fifth, the estimates show that the Elevate Math summer program helped reduce summer 
learning loss and increase achievement through the first half of the summer. However, 
whether the gains that were seen in the Elevate Math summer program persist into the 
fall or have broader effects on student math learning in subsequent years is not yet known. 
Further research involving longer follow-up and a control group with no access to the 
Elevate Math summer program would help answer that question. 


Whether the gains 
that were seen in 
the Elevate Math 
summer program 
persist into the fall 
or have broader 
effects on student 
math learning in 
subsequent years 
is not yet known 
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Appendix A. Data, outcomes, and methodology 


This appendix provides details on the study data, outcomes, and methodology. 

Data 

The study’s randomized controlled trial was conducted at eight middle school sites in six 
districts in California’s Silicon Valley: Alum Rock Union Elementary School District, Ben 
ryessa Union Elementary School District, Evergreen School District, Morgan Hill Unified 
School District, Mount Pleasant School District, and Oak Grove School District. Students 
typically attended the Elevate Math summer program on the same campus they attended 
during the school year, though a subset of students attended at nearby campuses. The dis- 
tricts are in suburban communities near San Jose, California. The districts’ enrollments 
range from 2,487 to 13,162, with an average of 9,426. The percentage of English learner stu- 
dents in each district ranges from 19 percent to 53 percent, with an average of 38 percent. 
The average percentage of students eligible for free or reduced-price lunch is 57 percent. 
And, on average, 52 percent of the students are Hispanic, 34 percent are Asian, 9 percent 
are White (non-Hispanic), and 2 percent are Black (notvHispanic). 

Recruitment efforts by the Silicon Valley Education Foundation yielded 496 students to 
participate in the study. Some 19 students opted out of data collection and were removed 
from the sample, leaving 477 students (figure Al). 

Because the Silicon Valley Education Foundation places a high priority on serving every 
student that is eligible and interested in participating in the program, randomly assigning 
students to a control group that would never receive access to the program was unde- 
sirable. Therefore, instead of randomly assigning students to a treatment group that was 
given access to the program and a control group that was not, students who were eligible 
and willing to participate were randomly assigned either to a treatment group that was 
scheduled to enroll in the Elevate Math summer program for the first four-week session 
or to a control group that did not enroll in the first four-week session but was allowed to 
enroll for the second four-week session. Random assignment was conducted separately for 
students in each of the school sites by Regional Educational Laboratory West staff, based 
on anonymized studentdevel data provided by the Silicon Valley Research Alliance. The 
random assignment process used Microsoft Excel and consisted of five steps: 

1. Establish the number of expected students in each of eight study sites. 

2. For each site, create a set of unique “slots” equal to the number of expected students 
plus 10. 

3. Create a random number variable and assign a random number to each of the slots. 

4. Determine the median of the random number variable and assign “session 1” status 
(for the treatment group) to all slots with values below the median and “session 2” 
status (for the control group) to all slots with values at or above the median. 

5. As students apply, place them in the next available slot and assign treatment status to 
the student based on the assigned status of the slot into which he or she is placed. 
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Figure Al. Student consolidated standards of reporting trials diagram 
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MDTP is Mathematics Diagnostic Testing Project. 

Source: Authors’ analysis of primary data collected for the study. 


Of the 477 students in the sample, 239 students were randomly assigned to the treatment 
group, and 238 students were randomly assigned to the control group. Seven students were 
assigned to attend the first session (treatment group) of the Elevate Math summer program 
but attended the second session, and fourteen students were assigned to the second session 
(control group) but attended the first session. These crossovers were kept in the study and 
were analyzed in the group to which they were originally assigned in the impact analy- 
ses. Of the 461 students who indicated their gender, 56 percent (n = 256) were male and 
44 percent (n = 205) were female. Based on their grade 6 math California Standards Test 
(CST) scores, more than half the students fell into the target range of the intervention: 
35 percent scored at the high basic level, and 18 percent scored at the low proficient level 
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(table Al). In addition, 29 percent scored below the target range for the intervention, and 
17 percent scored above the range. 

On the first day of each session, students took the Mathematics Diagnostic Testing Project 
(MDTP) test and then a perceptions survey. 4 They had 45 minutes to complete the test. 
There was no time limit for the survey, and most students spent 15 minutes or less on it. 
Teachers were instructed by the study team to provide a chance for a makeup test and 
survey for students who did not show up on the first day but came on the second day. The 
same test and survey administration was employed on the last day of the program. Teach- 
ers contacted students who were not present on the last day to offer a makeup test and 
survey the next day. 

Students for whom outcome data were obtained constituted the analytic samples for those 
outcomes. MDTP data were obtained for 349 students: 165 students in the treatment group 
and 184 in the control group. Survey measures of math interest and math self-efficacy 
were obtained from 159 students in the treatment group and 170 students from the control 
group. Baseline grade 6 math CST scores were obtained for 164 members of the treatment 
group follow-up sample and 181 members of the control group follow-up sample. 

Before conducting any formal analyses, a series of descriptive analyses were conducted to 
ensure the quality and accuracy of the data. These analyses examined whether the sample 
size (overall and by study site) was consistent with expectations, whether the MDTP test 
scores (overall and by topic area) were outside the range of expected values, whether the 
survey responses were aligned with the response scales, and whether the score distribu- 
tion was reasonable by comparing the study data to existing data collected previously 
for another purpose. Corrections were made during this data cleaning and examination 
process. The score distribution seemed to be similar to previously collected data. 

Overall data attrition was 27 percent, as data were collected from 73 percent of students 
(349 out of 477). The differential attrition between the treatment group and the control 
group was 8 percentage points (31 percent for the treatment group and 23 percent for the 
control group). 

To examine how data attrition might have affected the baseline equivalence between the 
treatment and control groups, the study team conducted the baseline equivalence test 
using grade 6 math CST scores (table A2). Though the differences were not statistically 


Table Al. Grade 6 math California Standards Test performance levels of study 
participants 


Basic Proficient 


Study 

participants 

Far below 
basic 

(scale 
score of 
150-252) 

Below 

basic 

(scale 
score of 
253-299) 

Low 

(scale 
score of 
300- 
324) 

High 

(scale 
score of 
325- 
349) 

Low 

(scale 
score of 
350- 
360) 

High 

(scale 
score of 
361- 
414) 

Advanced 

(scale 
score of 
415-600) 

Unknown 

Percent 

0.84 

8.60 

19.92 

34.80 

17.82 

14.05 

2.52 

1.47 

Number 
(n = 477) 

4 

41 

95 

166 

85 

67 

12 

7 


Source: Authors' analysis of primary data collected for the study. 
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Table A2. Baseline differences in grade 6 math California Standards Test scores 
were not statistically significant, regardless of which sample was used 


Study sample 



Adjusted mean 



Treatment 

group 

(standard 

deviation) 

Control 

group 

(standard 

deviation) 

Difference 

(standard 

error 3 ) 

p-value 

95 percent 
confidence 
interval 

Unweighted 

student 

sample 

size b 

Final analytic 

339.94 

333.20 

6.74 


-0.64 to 


sample 

(38.09) 

(29.60) 

(3.76) 

0.073 

14.13 

345 

Original sample 







for random 

340.42 

336.72 

3.71 


-2.64 to 


assignment 

(39.07) 

(32.11) 

(3.23) 

0.252 

10.05 

470 


Note: A regression model that accounted for study design characteristics (school site) was used to test the 
baseline equivalence between the treatment group and control group. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. The final analytic sample that was used for the impact analyses and with nonmissing grade 6 math Califor- 
nia Standards Test scores. 

Source: Authors' analysis of primary data collected for the study. 


significant, in the final analytic sample the point estimate for the treatment group average 
was 6.74 points above that for the control group. Examining the differences in the full 
random assignment sample shows that, while again the differences were not statistically 
significant, prior to any attrition the point estimate of average achievement for the treat- 
ment group is 3.71 points above that for the control group. Thus, 55 percent of the observed 
difference in the analytic sample was present at random assignment. In other words, most 
of the differences in the analytic sample were due to random variation produced by the 
random assignment process. The rest of the differences are due to nonrandom patterns 
in attrition: students with higher baseline CST scores were more likely to drop out of the 
control group sample than out of the treatment group sample. While this has the potential 
to introduce bias into the estimated effects, the impact estimates in this study control for 
baseline levels of achievement, so differences in outcomes would have to be due to factors 
that were correlated with attrition and math outcomes but not with baseline CST scores. 

Other study data that are required by the What Works Clearinghouse reporting guides 
regarding sample equivalency at the baseline are reported in tables A3 and A4 and data on 
the impact analyses on all outcome measures is reported in table A5. 

Outcomes 

The study examined the Elevate Math summer program’s effects on three sets of outcomes: 
math achievement and algebra readiness, achievement in specific math topic areas that 
appear to be aligned with the program’s described curriculum, and math interest and math 
self-efficacy. 

Math achievement was measured using the MDTP Algebra Readiness test, which consists 
of 45 multiple-choice items in seven topic areas (table A6). For each topic, the developer of 
the MDTP Algebra Readiness test has designated a “critical level,” which is the minimum 
number of correct responses required for a student to show adequate preparation in the 
topic. Previous research (Huang et al., 2014) has shown the MDTP Algebra Readiness test 
to be highly predictive of success in algebra in grade 8. In particular, seven MDTP Algebra 
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Table A3. Pre-intervention sample sizes and characteristics for the baseline sample 


Treatment group 

Control group 

Sample size Sample characteristics 

Sample size Sample characteristics 


Baseline measure 

Unit of 
assignment 

Unit of 
analysis 

Mean 

(adjusted 

mean) 

Standard 

deviation 

Unit of 
assignment 

Unit of 
analysis 

Mean 

(adjusted 

mean) 

Standard 

deviation 

Grade 6 math California 
Standards Test 

239 

236 

340.46 

(340.42) 

39.07 

238 

234 

336.68 

(336.72) 

32.11 


Note: A regression model with a dichotomous indicator for treatment status was used in the study to test for baseline equivalence 
between the treatment and control groups. Because random assignment was conducted separately within each school site, the model 
also included dichotomous indicators representing each school site. The standard error was estimated using the Huber-White proce- 
dure (Greene, 2003). The study sample (470 students) is the baseline sample with nonmissing grade 6 math California Standards Test 
scores. 

Source: Authors' analysis of primary data collected for the study. 


Table A4. Pre-intervention sample sizes and characteristics for the analytic sample 


Treatment group 

Control group 

Sample size Sample characteristics 

Sample size Sample characteristics 


Baseline measure 

Unit of 
assignment 

Unit of 
analysis 

Mean 

(adjusted 

mean) 

Standard 

deviation 

Unit of 
assignment 

Unit of 
analysis 

Mean 

(adjusted 

mean) 

Standard 

deviation 

Grade 6 math California 
Standards Test 

239 

164 

339.93 

(339.94) 

38.09 

238 

181 

333.20 

(333.20) 

29.60 


Note: A regression model with a dichotomous indicator for treatment status was used in the study to test for baseline equivalence 
between the treatment and control groups. Because random assignment was conducted separately within each school site, the model 
also included dichotomous indicators representing each school site. The study sample (345 students) is the final analytic sample that 
was used for the impact analyses and with nonmissing grade 6 math California Standards Test scores. 

Source: Authors' analysis of primary data collected for the study. 


Readiness topic area scores alone can predict Algebra I CST proficiency with 77 percent 
accuracy. 5 In addition, the higher the MDTP Algebra Readiness topic area score and the 
more MDTP Algebra Readiness topic areas a student has mastered, the higher the prob- 
ability that the student scores proficient or higher on the Algebra I CST in grade 8. A 
student who masters five or more topic areas has a 75 percent chance of scoring proficient 
or higher on the Algebra I CST in grade 8. 

Math achievement was defined as the MDTP Algebra Readiness total score. Algebra read- 
iness was defined according to whether students passed three or more topic areas on the 
MDTP Algebra Readiness test (based on findings by Huang et al., 2014, that grade 7 stu- 
dents who do so have a greater than 50 percent chance of succeeding in algebra I — that is, 
scoring proficient or higher on the Algebra I CST in grade 8. 

The three topic areas of the MDTP Algebra Readiness test that most closely aligned with the 
Elevate Math summer program’s curriculum were decimals, their operations and applications, 
and percent; literals and equations; and geometric measurement and coordinate geometry. 

The math interest and math self-efficacy measures were intended to capture two specif- 
ic outcomes, respectively: students’ interest in math and students’ self-confidence in their 
ability to do math. These dimensions were assessed using measures drawn primarily from 






Table A5. Post-intervention outcomes for the analytic sample and estimated effects 



Treatment group 

Control group 

Estimated effect 

Outcome measures 

Adjusted 

mean 

Standard 

deviation 

Adjusted 

mean 

Standard 

deviation 

Mean 

difference 

p-value 

(adjusted 

p-value) a 

Research question 1 

MDTP Algebra Readiness total scale 
score (points) 

20.81 

6.63 

16.80 

5.92 

4.01*** 

<.001 

(<.001) 

Algebra readiness (percent passing at 
least three of seven topic areas) 

29 

46 

12 

32 

0.17*** 

<.001 

(<.001) 

Research question 2 

MDTP score for decimals, their 
operations and applications, and percent 

3.91 

1.70 

3.05 

1.69 

0.86*** 

<.001 

(<.001) 

MDTP score for literals and equations 

3.65 

1.51 

3.13 

1.49 

0.52*** 

<.001 

(<.001) 

MDTP score for geometric measurement 
and coordinate geometry 

2.50 

1.44 

1.90 

1.29 

0.59*** 

<.001 

(<.001) 

Research question 3 

Math interest (1, low, to 5, high) 

2.83 

1.12 

2.63 

1.12 

0.20 

0.10 

(0.20) 

Math self-efficacy (1, low, to 5, high) 

3.21 

0.90 

3.19 

0.86 

0.02 

0.80 

(0.80) 


MDTP is Mathematics Diagnostic Testing Project. 

*** Significantly different from zero at the .001 level, two-tailed test. 

Note: A regression model with a dichotomous indicator for treatment status was used to estimate average differences between the 
treatment and control group outcomes. The model included prior achievement, as measured by grade 6 California Standards Test math 
scores, and dichotomous indicators representing each school site. The missing-indicator method (White & Thompson, 2005) was used 
to account for missing values on the grade 6 math CST. The standard errors used as the basis for statistical significance calculations 
were estimated using the Huber-White procedure (Greene, 2003). 

a. For the confirmatory research questions (1 and 3), the Benjamini-Hochberg (1995) procedure was used to calculate adjusted 
p-values across the two outcome measures. 

Source: Authors’ analysis of survey data collected by the Silicon Valley Education Foundation for the study. 


Table A6. Number of items and critical level by topic area on the Mathematics 
Diagnostic Testing Project Algebra Readiness test 

1 Topic area 

Number of items 

Critical level 1 

Integers 

6 

4 

Fractions and their applications 

8 

6 

Decimals, their operations and applications, and percent 

8 

6 

Exponents and square roots and scientific notation 

4 

3 

Literals and equations 

7 

5 

Geometric measurement and coordinate geometry 

6 

4 

Data analysis, probability, and statistics 

6 

4 

Source: California State University/University of California Mathematics Diagnostic Testing Project, 1995. 
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a student perception survey developed by Gilbert (2008) and administered by the Silicon 
Valley Education Foundation as part of its previous and ongoing program monitoring 
efforts. 

The math interest scale consisted of five items related to students’ interest in math (for 
example, how exciting math is to the students, how much the students like doing math). 
Each item asked students to rate their responses on a five-interval scale ranging from 1 
(not at all true for me) to 5 (very true for me), with the midpoint (3) indicating somewhat 
true for me. 

The math self-efficacy scale consisted of seven items related to students’ confidence in 
their ability to do math. For example, one item asks, “Even if a new topic in math is hard, 
how confident are you that you can learn it?” The response options are on a scale from 
1 (not at all confident) to 5 (very confident), with the midpoint (3) indicating somewhat 
confident. 

Previous research indicates that these survey scales are valid, reliable measures of student 
math interest and math self-efficacy, each with a Cronbach’s alpha that exceeds .7 (Gilbert, 
2008). In the study sample the Cronbach’s alpha for each survey scale (measured either in 
the pre-test or in the post-test) exceeded .9. 

Methodology 

Analytic approach and statistical adjustments. For all three research questions, analysis 
of covariance, a single-level regression model, was used to estimate program effects. A 
baseline measure (grade 6 math CST scores) was included as a covariate in the model 
to increase the precision of the estimate. Those CST scores were also used to examine 
baseline equivalence between the treatment and control groups. The model takes the fol- 
lowing form: 


DV.. = a n + B.CST6. + B,Tx.. + Sv Stratum + e.. 

ij U I 1 i 1 L i] o j ij 


where i denotes a student, j denotes a school site, DV is the outcome variable being studied, 
CST6 is the baseline measure and serves as a covariate, Tx is a dichotomous variable indi- 
cating students’ intervention condition (treatment or control), Stratum is a vector of fixed 
effects for j— 1 strata, and s is an error term for individual student members. The program 
effect is represented by (J 2 , which captures differences between the treatment and control 
groups on the outcome variable after controlling for baseline difference and study design 
factors. Because observations i are clustered within schools, the effect estimates account 
for the effects of clustering by estimating Huber-White standard errors (Greene, 2003), 
which avoid the downward bias on the standard errors that clustering can otherwise cause. 
Effect sizes were calculated by dividing effect estimates by the control group standard devi- 
ation of the outcome variable. 

Treatment of missing data. The missing-indicator method (White & Thompson, 2005) 
was used to account for missing values on the covariate (not the outcome variables) in 
the impact analysis models. With the missing-indicator method, all observations with 
missing values on covariates are retained in the analysis. Indicator variables are created 
for missing values on each variable (0 = observed, 1 = missing), and missing values on the 


A-7 


covariates are coded to a constant. Both the recoded covariates and the missing value 
indicator variables are included in the regression model. In a randomized controlled trial, 
in which randomization helps ensure that the baseline covariates are balanced, the use 
of the missing-indicator method appears to refine the precision of impact estimates and 
standard errors (White & Thompson, 2005). 

Observations with missing values on outcome variables were excluded from the impact 
analyses. Deletion of observations with missing outcome variables has been shown to result 
in accurate impact estimates and standard errors when outcomes are missing at random, 
conditional on the covariates (Allison, 2002; von Hippel, 2007). 

To examine how robust the findings are with respect to procedures for handling missing 
data, sensitivity analyses were conducted based on the sample with nonmissing post-test 
data and nonmissing data on the covariate. These findings are presented in appendix B. 

For the MDTP Algebra Readiness test, missing item responses were treated as incorrect 
responses. For the survey scales, scores were obtained from nonmissing item responses 
(that is, the scale score for each student was obtained by computing the average score over 
nonmissing item responses in that scale). For the math interest scale, fewer than 5 students 
had one or more missing item responses; for the math self-efficacy scale, fewer than 13 
students had one or more missing item responses. 

Multiple hypothesis testing. The confirmatory research questions were research questions 
1 and 3. The questions address two domains: content knowledge in math and attitudes 
toward math and math learning. The impact analysis in the content knowledge domain 
focuses on two outcomes: math achievement and algebra readiness. As such, multiple com- 
parison procedures were applied to reduce the probability of finding statistically signifi- 
cant program effects within that domain due to chance factors alone. Similarly, the math 
attitudes domain has two outcomes (math interest and math self-efficacy), and multiple 
comparison procedures were used to make adjustments to the statistical significance tests 
within that domain as well. No across-domain adjustment was made. 

Benjamini and Hochberg’s (1995) stepwise multiple hypothesis testing procedure was used 
to test impact estimates for cognition domain. This procedure involves ordering p -values 
obtained for each outcome variable from largest to smallest, multiplying each unadjusted 
p-value by N/(N - j + 1), where N is the number of outcome variables within a domain and 
j is the order of the test. The procedure involves rejecting all null hypotheses in which the 
adjusted p-value is less than .05. 

Treatment of crossovers. Because this study is based on an intent to treat analysis (Bloom, 
2006), crossover students were analyzed with their original random assignment group, irre- 
spective of whether they participated in the first or second session of the program. The 
study team conducted a sensitivity analysis to examine the effects of crossovers on the 
estimated program effects. In particular, the data were reanalyzed, treating treatment-to- 
control crossovers as control students and control-to-treatment crossovers as treatment stu- 
dents. Findings of these sensitivity analyses are in appendix B. The results of this analysis 
suggest a change in the MDTP impact estimate from 4.0 points to 3.6. 


A-8 


The analysis also included an estimate of the local average treatment effect to assess the 
effect of the treatment on those who actually participated (Bloom, 2006). In particular, 
the estimated effect on MDTP achievement was divided by the difference in participation 
rates between the treatment and control groups. In this case, participation was defined 
by participation in the first session. Based on the analytic sample, 158 out of 165 students 
from the treatment group (96 percent) participated in the treatment (session 1), while 14 
out of 184 (8 percent) students from the control group also participated in the treatment 
(session 1). Dividing the effect on MDTP achievement of 4 points by the difference in 
these rates (88 percentage points), suggests a local average treatment effect of 4.5 points. 
Together, these analyses suggest that the presence of crossovers did not materially change 
the pattern of effects. 
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Appendix B. Sensitivity analyses 


For research question 1 (program effect on math achievement), the study team conducted 
several sensitivity analyses to study the robustness of program impact estimate shown in 
table 1 in the main report. The reported program impact estimate using the MDTP total 
score is based on model 2b in table Bl; the reported program impact estimate of algebra 
readiness is based on model 2bl in table B2. Findings of these sensitivity analyses indicate 
that the reported estimate was stable and not moving around by model specification. 

Similar sensitivity analyses were conducted for program effects on math interest and math 
self-efficacy (tables B3 and B4). Findings of these sensitivity analyses suggest that the 
reported program impact estimates as shown in table 4 in the main text are robust, regard- 
less of how the impact model is specified. 


Table Bl. Elevate Math summer program effect on grade 7 students’ math Mathematics Diagnostic 
Testing Project total scale score, by model specification 


Model 

Adjusted mean 

Treatment Control 

group group 

(standard (standard 

deviation) deviation) 

Difference 

(standard 

error 3 ) 

p-value 

(adjusted 

p-value b ) 

95 percent 
confidence 
interval 

Effect size c 

Unweighted 
student 
sample size 

1 

21.14 

16.50 

4.64*** 

<.001 





(6.63) 

(5.92) 

(0.68) 

(c.001) 

3.30 to 5.97 

0.78 

349 

2a 

20.76 

16.77 

3.99*** 

<.001 





(6.64) 

(5.90) 

(0.57) 

(c.001) 

2.88 to 5.11 

0.68 

345 

2b 

20.81 

16.80 

4.01*** 

<.001 





(6.63) 

(5.92) 

(0.56) 

(c.001) 

2.90 to 5.12 

0.68 

349 

2c 

20.56 

16.92 

3.64*** 

<.001 





(6.50) 

(6.16) 

(0.57) 

(c.001) 

2.52 to 4.76 

0.59 

349 

3a 

20.78 

16.72 

4.06*** 

<.001 





(6.54) 

(5.75) 

(0.56) 

(<.001) 

2.97 to 5.16 

0.71 

340 

3b 

20.78 

16.83 

3.95*** 

<.001 





(6.63) 

(5.92) 

(0.56) 

(c.001) 

2.84 to 5.05 

0.67 

349 


*** Significantly different from zero at the .001 level, two-tailed test. 

Note: Data were regression-adjusted to account for differences in baseline measures or study design characteristics (strata). Model 1 
includes no covariates, only strata. Model 2a includes grade 6 math California Standards Test scores as a covariate using the sample 
with both nonmissing outcome variable and nonmissing covariate. Model 2b is similar to model 2a, but the missing-indicator method 
(White & Thompson, 2005) was used to account for missing values on the covariate; this is the model used to report the impact esti- 
mate in table 2 in the main text. Model 2c is similar to model 2b, but the treatment status was changed for those crossovers. Model 3a 
is similar to model 2a but adds gender as another covariate. Model 3b is similar to model 2b but adds gender as another covariate. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. The Benjamini-Hochberg (1995) procedure was used to calculate adjusted p-values across the two outcome measures. 

c. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

Source: Authors’ analysis of primary data collected for the study. 
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Table B2. Elevate Math summer program effect on grade 7 students’ algebra readiness, by model 
specification 


Model 

Adjusted mean 

Treatment Control 

group group 

(standard (standard 

deviation) deviation) 

Difference 

(standard 

error 3 ) 

p-value b 

(adjusted 

p-value c ) 

95 percent 
confidence 
interval 

Effect size d 

Unweighted 
student 
sample size 

1 

0.31 

0.11 

0.19*** 

<.001 





(0.46) 

(0.32) 

(0.04) 

(<.001) 

0.11 to 0.28 

0.59 

349 

2a 

0.29 

0.12 

0.17*** 

<.001 





(0.46) 

(0.31) 

(0.04) 

(<.001) 

0.09 to 0.25 

0.55 

345 

2bl 

0.29 

0.12 

0.17*** 

<.001 





(0.46) 

(0.32) 

(0.04) 

(<.001) 

0.09 to 0.25 

0.53 

349 

2b2 

0.30 

0.11 

0.19*** 

<.001 





(0.46) 

(0.32) 

(0.04) 

(<.001) 

0.10 to 0.27 

.59 

349 

2c 

0.28 

0.13 

0 . 14 *** 

<.001 





(0.45) 

(0.33) 

(0.04) 

(<.001) 

0.07 to 0.22 

0.42 

349 

3a 

0.28 

0.11 

0.17*** 

<.001 





(0.46) 

(0.30) 

(0.04) 

(<.001) 

0.09 to 0.25 

0.57 

340 

3b 

0.29 

0.12 

0.17*** 

<.001 





(0.46) 

(0.32) 

(0.04) 

(<.001) 

0.09 to 0.25 

0.53 

349 


*** Significantly different from zero at the .001 level, two-tailed test. 

Note: Data were regression-adjusted to account for differences in baseline measures or study design characteristics (strata). Model 1 
includes no covariates, only strata. Model 2a includes grade 6 math California Standards Test scores as a covariate using the sample 
with both nonmissing outcome variable and nonmissing covariate. Model 2bl is similar to model 2a, but the missing-indicator method 
(White & Thompson, 2005) was used to account for missing values on the covariate; this is the model used to report the impact 
estimate in table 2 in the main text. Model 2b2 is the same as model 2bl, but a logistic regression analysis was used, treating algebra 
readiness as a dichotomous variable. Model 2c is similar to model 2bl, but the treatment-to-control crossovers were treated as stu- 
dents in the control group, and control-to-treatment crossovers were treated as students in the treatment group. Model 3a is similar to 
model 2a but adds gender as another covariate. Model 3b is similar to model 2bl but adds gender as another covariate. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. Represents the significance of the test that is associated with the estimated odds ratio for the treatment or control indicator. 

c. The Benjamini-Hochberg (1995) procedure was used to calculate adjusted p-values across the two outcome measures. 

d. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

Source: Authors’ analysis of primary data collected for the study. 
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Table B3. Elevate Math summer program effect on grade 7 students’ math interest, by model 
specification 


Model 

Adjusted mean 

Treatment Control 

group group 

(standard (standard 

deviation) deviation) 

Difference 

(standard 

error 3 ) 

p-value 

(adjusted 

p-value b ) 

95 percent 
confidence 
interval 

Effect size 3 

Unweighted 
student 
sample size 

1 

2.84 

2.62 

0.22 

0.07 





(1.12) 

(1.12) 

(0.12) 

(0.14) 

-0.02 to 0.45 

0.20 

329 

2a 

2.81 

2.63 

0.17 

0.14 





(1.12) 

(1.11) 

(0.12) 

(0.28) 

-0.06 to 0.41 

0.15 

324 

2b 

2.83 

2.63 

0.20 

0.10 





(1.12) 

(1.12) 

(0.12) 

(0.20) 

-0.04 to 0.43 

0.18 

329 

2c 

2.82 

2.63 

0.20 

0.09 





(1.09) 

(1.15) 

(0.12) 

(0.18) 

-0.03 to 0.43 

0.17 

329 

3a 

2.81 

2.63 

0.18 

0.14 





(1.10) 

(1.14) 

(0.12) 

(0.28) 

-0.06 to 0.41 

0.16 

319 

3b 

2.82 

2.64 

0.19 

0.11 





(1.12) 

(1.12) 

(0.12) 

(0.22) 

-0.04 to 0.42 

0.17 

329 


Note: Data were regression-adjusted to account for differences in baseline measures or study design characteristics (strata). Model 1 
includes no covariates, only strata. Model 2a includes grade 6 math California Standards Test scores as a covariate using the sample 
with both nonmissing outcome variable and nonmissing covariate. Model 2b is similar to model 2a, but the missing-indicator method 
(White & Thompson, 2005) was used to account for missing values on the covariate; this is the model used to report the impact esti- 
mate in table 5 in the main text. Model 2c is similar to model 2b, but the treatment status was changed for those crossovers. Model 3a 
is similar to model 2a but gender was added as another covariate. Model 3b is similar to model 2b, but gender was added as another 
covariate. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. The Benjamini-Hochberg (1995) procedure was used to calculate adjusted p-values across the two outcome measures. 

c. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

Source: Authors' analysis of primary data collected for the study. 
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Table B4. Elevate Math summer program effect on grade 7 students’ math self-efficacy, by model 
specification 


Model 

Adjusted mean 

Treatment Control 

group group 

(standard (standard 

deviation) deviation) 

Difference 

(standard 

error 3 ) 

p-value 

(adjusted 

p-value b ) 

95 percent 
confidence 
interval 

Effect size 3 

Unweighted 
student 
sample size 

1 

3.23 

3.17 

0.05 

0.60 





(0.90) 

(0.86) 

(0.10) 

(0.60) 

-0.13 to 0.24 

0.06 

329 

2a 

3.21 

3.19 

0.01 

0.88 





(0.91) 

(0.84) 

(0.09) 

(0.88) 

-0.17 to 0.20 

0.01 

324 

2b 

3.21 

3.19 

0.02 

0.80 





(0.90) 

(0.86) 

(0.09) 

(0.80) 

-0.16 to 0.21 

0.03 

329 

2c 

3.23 

3.17 

0.06 

0.51 





(0.90) 

(0.86) 

(0.09) 

(0.51) 

-0.12 to 0.25 

0.07 

329 

3a 

3.20 

3.19 

0.01 

0.89 





(0.91) 

(0.84) 

(0.09) 

(0.89) 

-0.17 to 0.20 

0.01 

319 

3b 

3.21 

3.19 

0.02 

0.85 





(0.90) 

(0.86) 

(0.09) 

(0.85) 

-0.17 to 0.20 

0.02 

329 


Note: Data were regression-adjusted to account for differences in baseline measures or study design characteristics (strata). Model 1 
includes no covariates, only strata. Model 2a includes grade 6 math California Standards Test scores as a covariate using the sample 
with both nonmissing outcome variable and nonmissing covariate. Model 2b is similar to model 2a, but the missing-indicator method 
(White & Thompson, 2005) was used to account for missing values on the covariate; this is the model used to report the impact esti- 
mate as shown in table 5 in the main text. Model 2c is similar to model 2b, but the treatment status was changed for those crossovers. 
Model 3a is similar to model 2a but adds gender as another covariate. Model 3b is similar to model 2b but adds gender as another 
covariate. 

a. Estimated using the Huber-White procedure (Greene, 2003). 

b. The Benjamini-Hochberg (1995) procedure was used to calculate adjusted p-values across the two outcome measures. 

c. Calculated by dividing impact estimates by the control group standard deviation of the outcome variable. 

Source: Authors' analysis of primary data collected for the study. 


B-4 




Notes 


1. The CST has five performance levels, which are based on scale scores: far below basic, 
below basic, basic, proficient, and advanced. High basic is defined as a scale score of 
325-349, and low proficient is defined as a scale score of 350-360. 

2. Initial algebra I course placement is typically based on grade 6 math CST score because 
grade 7 math CST scores are not available until late August or early September. 

3. This includes the Alum Rock, Berryessa, Franklin-McKlinley, Oak Grove, and San 
Jose school districts. 

4. The correlation between the test scores and the survey scale scores is low: less than 
0.2 (for both the interest scale and the self-efficacy scale) for students in the treat- 
ment group who took the post-test and 0.14 (with the interest scale) and 0.27 (with the 
self-efficacy scale) for students in the control group who took the pre-test. This indi- 
cates that the effect on the survey scales after taking the test is minimal and is about 
the same between treatment and control groups. 

5. Accuracy is defined with respect to the students who were placed into algebra I in 
grade 8. It refers to the percentage of these students who are predicted to succeed in 
algebra I in grade 8 who actually score proficient or higher on the Algebra I CST in 
the following school year. This percentage is also called the positive predictive value. 
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